Perfect Synthesis for All of the People All of the Time

The quality of speech synthesis has drastically improved over the last ten years. Or at least it appears that this is the case. We have moved from diphones to unit selection. However, although we can produce much more natural sounding examples we have also given up an certain amount of control over what can be synthesized. We have reached the stage where playing a few examples to a non-expert can easily convince them that speech synthesis is a solved problem. This paper looks at how we might not only convince some of the people some of the time, but what we must do to produce perfect synthesis for all of the people all of the time. 1. UNIT SELECTION SYNTHESIS The basic unit selection premise is that we can synthesize new naturally sounding utterances by selecting appropriate sub-word units from a database of natural speech. There are lots of conditions that must be met in order for such a system to work. Let us consider the following basic notion of unit selection. Although this particular instantiation comes from [1] it will be generalized to help illustrate the space of the problems. In [1] and in later, and earlier unit selection techniques [2], there is a notion of a target cost, how close a database unit is to desired unit, and a join cost, how well two adjacently selected units join together. The unit selection process is designed to optimally minimise both target and join costs. More formally we can define the target cost to be the weighted sum of differences of relevant features. Various features have been proposed, typically encoding phonetic, metrical, and prosodic context. In addition to selecting based on target cost we can define continuity cost as a weighted sum of difference of features